IML Unit 19 - Neural Networks Regularization

Introduction

This unit continues our exploration of neural networks by focusing on techniques for improving their generalization, designing suitable architectures, and training them efficiently.

                Today's Focus:
                Regularization: Dropout, L1/L2, Early Stopping
Architecture Design: Choosing layers, units, and hyperparameters
Optimization: Learning rate strategies, SGD variants

            

This lecture builds upon the introduction to the backward pass in Unit 21 and focuses on how to design effective neural networks and train them efficiently.

2. Theory

Observations:

Logistic Regression: Creates linear decision boundaries and is limited to linearly separable problems.
Decision Tree: Creates piecewise linear boundaries. It can handle some nonlinearity but may overfit.
Gradient Boosting: Creates complex, smooth boundaries. It can be very effective but may be slow.
Neural Network: Can learn highly complex, nonlinear boundaries. It is highly flexible but requires careful tuning. Taken together, these observations show why neural networks offer greater flexibility but also require careful design and tuning.

Important: A neural network may need a more suitable configuration for a complex problem. A single hidden layer with 10 units may not be sufficient, so we will examine how architecture and training choices can improve the model.

2.1 Regularization Methods

Regularization methods are techniques used to prevent overfitting and improve the generalization of neural networks.

How they work: They introduce constraints or penalties on model parameters so that the model does not become unnecessarily complex or fit noise in the training data. The common goal is to improve generalization by discouraging an overly complex model.

In neural networks, three widely used regularization techniques are:

Dropout
L1 / L2 Regularization
Early Stopping

2.2 Dropout

Dropout randomly drops a subset of hidden units during training, so different subsets of units participate in each training iteration.

                How Dropout Works:
                At each training iteration, each hidden node is independently assigned a Bernoulli random variable:
                        1 → keep the node
0 → drop the node

                    
Dropped nodes do not participate in the forward pass (their outputs are zeroed)
In the backward pass, their weights are not updated
Thus, every training iteration effectively uses a different, randomly-thinned network

            

Dropout Rate:

The fraction of units dropped at each training iteration
Typical values: 0.1-0.5, and rarely higher
Example (Keras): model.add(Dropout(0.25))

Important: Dropout is only active during training, not during inference (prediction). At test time, all units are used, but their outputs are scaled by the dropout rate to maintain expected output magnitudes.

2.3 L1/L2 Regularization

We already covered ridge, lasso, and elastic net in regression. The same regularization idea also applies to neural networks:

L2 Regularization (Ridge):Effect: Encourages small, diffuse weights, which can lead to smoother functions
The sum runs over all weights in all layers. Biases are usually excluded.
If the original loss is \(L(y, \hat{y})\), then with L2 regularization:

                    \[
                    \mathcal{J}(x, y, w, b) = \mathcal{L}(y, \hat{y}) + \lambda \sum_{i} w_{i}^{2}
                    \]
                

L1 Regularization (Lasso):Effect: Encourages sparse weights, so some weights can become exactly zero

                    \[
                    \mathcal{J}(x, y, w, b) = \mathcal{L}(y, \hat{y}) + \lambda \sum_{i} |w_{i}|
                    \]
                

Note: In neural networks, L2 is used far more commonly than L1.

2.4 Early Stopping

Early stopping stops training before the model begins to overfit.

                How Early Stopping Works:
                Split the data into training and validation sets
During training, monitor the validation loss.
If the validation loss stops improving (e.g., for 5 epochs), terminate training

            

Interpretation:

Early stopping is effectively a regularizer on the number of training steps
Training for too long can lead to overfitting; stopping earlier keeps the model in a "simpler" region of parameter space

2.5 Classification Example

Consider a classification problem with the following features:

Obs.	ALCHL_I	PROFIL_I_R	SUR_COND	VEH_INVL	MAX_SEV_IR
1	1	1	1	1	1
2	2	1	1	1	0
3	2	1	1	1	1
4	1	1	1	1	0
5	2	1	1	1	2
6	2	0	1	1	1
7	2	0	1	3	1
8	2	0	1	4	1
9	2	0	1	2	0
10	2	0	1	2	0

Feature	Description
ALCHL_I	Presence (1) or absence (2) of alcohol
PROFIL_I_R	Profile of the roadway: level (1), other (0)
SUR_COND	Surface condition of the road: dry (1), wet (2), snow/slush (3), ice (4), unknown (9)
VEH_INVL	Number of vehicles involved
MAX_SEV_IR	Presence of injuries/fatalities: no injuries (0), injury (1), fatality (2)

To use a neural network for this classification problem:

Use 7 nodes in the input layer (one for each predictor)
Use 3 neurons in the output layer (one for each class)
Use a single hidden layer and experiment with the number of nodes
Increase the number of hidden nodes from one to five and examine the resulting confusion matrices to identify a size that improves training performance without deteriorating validation performance

2.6 Guidelines for Choosing Architecture

For tabular data, 1-2 hidden layers are often sufficient:

                Universal Approximation Theorem: A single hidden layer can represent complex non-linear relationships between predictors
Size of hidden layers: The number of nodes determines the network's capacity:
                        Too few nodes: → underfitting (cannot capture the required complexity)
Too many nodes: → overfitting (may memorize the training data)

Rule of thumb for tabular data:

Start with p to 2p nodes (where p = number of input features)
Or try common sizes: 32, 64, 128 nodes
Monitor validation performance and adjust the architecture as needed
Use regularization techniques (dropout, early stopping) to control overfitting

Choosing an Architecture (Continued)

                Number of output nodes:
                For classification (categorical outcome with m classes):
                        Use m nodes with softmax activation (most common)
Or m-1 nodes (the m-th class probability is implicit)

Special case — Binary classification:
                        Often use 1 node with sigmoid activation

For regression (numerical outcome):
                        Use 1 node with linear activation (no activation function)
Use k nodes if predicting k different numerical targets simultaneously (multi-output regression)

2.7 Learning Rate

The learning rate controls how much the weights are adjusted during each update. Choosing an appropriate strategy helps achieve efficient and stable training.

Strategy 1: Fixed Learning Rate

                Description: Keep the learning rate constant throughout training (e.g., \(\eta = 0.001\))
Advantage: Simple, no tuning needed
Disadvantage: It may be too large, causing oscillation around the minimum, or too small, leading to slow convergence.

            

Strategy 2: Learning Rate Decay/Scheduling

Description: Start with a larger value (\(\eta_0\)) and gradually decrease it over time.

Rationale: Learn quickly initially, then fine-tune the weights as training progresses.

Common schedules:

Step decay: Reduce the learning rate by a factor (e.g., ÷5) every N iterations
Exponential decay: \(\eta = \eta_0 \cdot e^{-kt}\)
1/t decay: \(\eta = \eta_0 / (1 + kt)\), where \(t =\) iteration number

Strategy 3: Adaptive/Performance-Based

                Description: Monitor the loss function during training.
Rule: As long as the loss is decreasing, keep the current learning rate.
When the loss plateaus (stops decreasing for a set number of iterations), reduce the learning rate (e.g., divide by 5 — sklearn default).
This allows network to escape plateaus and find better solutions

            

Strategy 4: Adaptive Optimizers (Modern Default)

                Description: Use optimizers that automatically adjust the learning rate for each parameter.
Examples: Adam, RMSprop, AdaGrad
Mechanism: Maintain different learning rates for each weight and adapt them based on gradient history.
Usage: These are among the most common choices in modern deep learning.

            

2.8 Weight Initialization

Initializing the weights and biases appropriately is important for convergence during training. Poor initialization can lead to slow convergence, poor solutions, or vanishing or exploding gradients.

                Zero Initialization: Setting all weights to zero is a common but not always suitable strategy. All neurons in a layer compute the same output and update identically, preventing the network from learning asymmetric features.
Random Initialization: Initialize the weights with small random values, usually drawn from a normal (Gaussian) or uniform distribution.

Xavier/Glorot Initialization:

Sets the weights using a normal distribution with a mean of 0 and a variance of \(2 / (\text{number of input and output units})\)
Effective for sigmoid and hyperbolic tangent (tanh) activation functions

He Initialization:

Similar to Xavier, but with a variance of \(2 / \text{number of input units}\)
Often used with rectified linear unit (ReLU) activation functions

2.9 Batch, Mini-Batch, and SGD

Different gradient descent variants affect how often parameters are updated and can change training efficiency and convergence:

                Batch (Full-Batch) Gradient Descent:
                        Computes gradients over the entire training set before updating weights and biases
Pros: Stable convergence, exact gradient
Cons: Computationally expensive for large datasets, requires loading all data into memory

                    
Stochastic Gradient Descent (SGD):
                        Update the parameters after each individual training example
Pros: The "noisy" updates can help escape local minima
Cons: The high variance in the gradient estimates may lead to slower convergence

                    
Mini-Batch SGD:
                        Use a subset (mini-batch) of the training data to compute the gradient and update the parameters
The mini-batch size is a hyperparameter (e.g., 32, 64, 128)
Pros: Balances the stability of batch gradient descent and the efficiency of SGD
Cons: Gradient estimates still contain some noise

                    

            

Training dynamics: The number of parameter updates per epoch depends on the gradient descent variant.

On each epoch (a full pass through data), parameters may be updated many times if using SGD or mini-batch
For SGD: Number of updates per epoch = number of training examples
For mini-batch SGD: Number of updates per epoch = number of batches

2.10 Momentum

Standard gradient descent can be slow in valleys (long, narrow regions) and can oscillate in steep directions.

Standard Gradient Descent:

\[ \theta_{\text{new}} = \theta_{\text{old}} - \eta \cdot \nabla L(\theta) \]

Momentum:

Momentum adds "inertia" to updates by accumulating past gradients, similar to a ball rolling downhill.

\[ v_{\text{new}} = \beta \cdot v_{\text{old}} + \nabla L(\theta) \] \[ \theta_{\text{new}} = \theta_{\text{old}} - \eta \cdot v_{\text{new}} \]

Benefits:

Speeds up convergence when gradient directions are consistent
Reduces oscillations and can help escape shallow local minima

Note: In practice, modern optimizers like Adam incorporate momentum-like mechanisms automatically, so you rarely need to implement it manually.

2.11 Neural Networks: Advantages and Disadvantages

Neural networks are powerful models but come with important tradeoffs. The most prominent advantage is their good predictive performance. They are known to have high tolerance to noisy data and the ability to capture highly complicated non-linear relationships between predictors and an outcome variable.

Their weakest point is in providing insight into the structure of the relationship, hence their blackbox reputation. Several considerations and dangers should be kept in mind when using neural networks:

Advantages

Disadvantages / Risks

Key Considerations

Excellent predictive performance on complex, non-linear patterns
High tolerance to noisy training data
Ability to model highly complicated feature interactions automatically
State-of-the-art results in vision, NLP, speech, and many structured tasks

Blackbox nature: Hard to explain why a prediction was made
Extrapolation danger: Predictions outside training range can be completely invalid
No built-in variable selection: Careful predictor preprocessing is required
Data-hungry: Flexibility relies heavily on having sufficient training data
Computationally expensive: Runtime grows greatly with number of predictors (more weights to compute)

In classification problems, the network requires sufficient records of the minority class — achieved via oversampling
Weight initialization, optimizer choice, batch size, and regularization strategy all matter significantly
Use validation splits and callbacks (early stopping) to monitor convergence and avoid overfitting

2.12 NN Training Summary (7-Step Workflow)

Select the architecture: Number of layers, their sizes, and the type of activation function.
Initialize weights and biases: Use intelligently selected initial values (e.g., He, Glorot).
Forward pass minibatch: Run a minibatch through the network and compute the mean loss.
Backpropagation: Calculate the contribution of each weight and bias to the overall loss for the minibatch.
Gradient descent update: Update the weight and bias values of the model based on the contributions.
Repeat: Continue from step 3 until desired epochs, threshold loss, or validation convergence.
Regularize if needed: Apply L1/L2, dropout, early stopping, or data augmentation if the network isn't learning well.

Try It Yourself

Problem 1: Activation Function Selection

You are building a neural network for each of the following tasks:

Predicting house prices (regression)
Binary classification (spam detection)
Multiclass classification (handwritten digit recognition)

Task: What activation function would you use for the output layer in each case?

Solution:

House price prediction (regression): Linear (no activation function)
Spam detection (binary classification): Sigmoid
Digit recognition (multiclass classification): Softmax

Problem 2: ReLU Derivative

Given the ReLU activation function \(f(z) = \max(0, z)\), calculate the derivative for the following inputs:

z = 2
z = -1
z = 0

Solution:

Using the derivative definition:

\[ \operatorname{ReLU}'(z) = \begin{cases} 1 & z > 0 \\ 0 & z \leq 0 \end{cases} \]

z = 2: Since 2 > 0, ReLU'(2) = 1
z = -1: Since -1 ≤ 0, ReLU'(-1) = 0
z = 0: Since 0 ≤ 0, ReLU'(0) = 0

Problem 3: Softmax Calculation

Calculate the softmax for the following input vector:

z = [1, 2, 3]

Task: Compute softmax(z)

Solution:

Using the softmax formula:

\[ \operatorname{softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^K e^{z_j}} \]

Step 1: Compute exponentials:

e^1 ≈ 2.718
e^2 ≈ 7.389
e^3 ≈ 20.086
Sum = 2.718 + 7.389 + 20.086 ≈ 30.193

Step 2: Compute softmax for each element:

softmax(1) = 2.718 / 30.193 ≈ 0.090
softmax(2) = 7.389 / 30.193 ≈ 0.245
softmax(3) = 20.086 / 30.193 ≈ 0.665

Verification: 0.090 + 0.245 + 0.665 ≈ 1.000 ✓

Problem 4: Dropout Implementation

You have a hidden layer with 100 neurons and want to apply dropout with a rate of 0.25.

Tasks:

How many neurons will be kept (on average) in each training iteration?
What is the probability that a specific neuron is dropped?
During inference (testing), if a neuron has an activation of 0.8, what will be its scaled output?

Solution:

Neurons kept: 100 × (1 - 0.25) = 75 neurons (on average)
Probability of dropping: 0.25 (dropout rate)
Scaled output during inference: At test time, dropout is turned off, but outputs are scaled by the dropout rate to maintain expected values. So 0.8 × (1 - 0.25) = 0.8 × 0.75 = 0.6

Problem 5: Learning Rate Scheduling

You are training a neural network with an initial learning rate of \(\eta_0 = 0.1\).

Tasks:

Using step decay with a factor of 0.5 every 100 iterations, what is the learning rate at iteration 250?
Using exponential decay with \(k = 0.01\), what is the learning rate at iteration 100?
Using 1/t decay with \(k = 0.1\), what is the learning rate at iteration 50?

Solution:

Step decay: At iteration 250, we've passed 2 decay points (100 and 200). Learning rate = 0.1 × (0.5)^2 = 0.1 × 0.25 = 0.025
Exponential decay: \(\eta = \eta_0 \cdot e^{-kt} = 0.1 \cdot e^{-0.01 \times 100} = 0.1 \cdot e^{-1} \approx 0.1 \times 0.3679 = \) 0.03679
1/t decay: \(\eta = \eta_0 / (1 + kt) = 0.1 / (1 + 0.1 \times 50) = 0.1 / (1 + 5) = 0.1 / 6 \approx \) 0.01667

Interactive Quiz

Test your understanding of Neural Networks Advanced Topics:

Score: 0 / 5

Key Takeaways

Regularization:

Dropout: Randomly drops neurons during training and helps prevent co-adaptation; typical rate: 0.1-0.5
L1 Regularization: Encourages sparse weights, some weights become exactly zero
L2 Regularization: Encourages small, diffuse weights and is more common in neural networks
Early Stopping: Stops training when validation loss stops improving and helps prevent overfitting

Architecture Design:

For tabular data: 1-2 hidden layers are typically sufficient
Hidden layer size: Start with p-2p nodes (p = input features) or try 32, 64, or 128
Output layer: Softmax for multiclass classification, sigmoid for binary classification, and linear for regression
For other data types: CNNs for images and Transformers for text

Optimization:

Learning rate strategies: Fixed, decay, adaptive, or adaptive optimizers such as Adam
Weight initialization: Xavier/Glorot for sigmoid/tanh, He for ReLU
Gradient descent variants: Batch (stable but slow), SGD (noisy but fast), and Mini-batch (a balance of the two)
Momentum: Adds inertia to updates, speeds up convergence, and reduces oscillations

Common Pitfalls

Regularization:

Using dropout in the output layer: Dropout should typically be applied only to hidden layers
Dropout rate too high: Can cause underfitting; a typical range is 0.1-0.5
Dropout during inference: Dropout should be turned off during testing and prediction
Early stopping too early: May stop before the model has learned useful patterns
Early stopping too late: May allow the model to overfit
L1/L2 regularization strength: A λ that is too large can cause underfitting, while a λ that is too small may not prevent overfitting

Architecture Design:

Too few hidden units: May not provide enough capacity to learn complex patterns, causing underfitting
Too many hidden units: May overfit the training data and slow training
Too many layers for simple problems: Add unnecessary complexity and may lead to overfitting
Not using regularization: Deep networks with many parameters are more prone to overfitting
Fixed architecture: Failing to experiment with different architectures can prevent you from finding a better configuration

Optimization:

Learning rate too large: Can cause weights to oscillate or diverge
Learning rate too small: Can lead to very slow convergence
Poor weight initialization: Can lead to slow convergence or getting stuck in poor local minima
Batch size too small: Can lead to noisy gradient estimates and slow convergence
Batch size too large: Can be memory-intensive and may slow training
Not using momentum: Can lead to slow convergence in valleys and oscillations in steep directions

Obs.	ALCHL_I	PROFIL_I_R	SUR_COND	VEH_INVL	MAX_SEV_IR
1	1	1	1	1	1
2	2	1	1	1	0
3	2	1	1	1	1
4	1	1	1	1	0
5	2	1	1	1	2
6	2	0	1	1	1
7	2	0	1	3	1
8	2	0	1	4	1
9	2	0	1	2	0
10	2	0	1	2	0

Obs.	ALCHL_I	PROFIL_I_R	SUR_COND	VEH_INVL	MAX_SEV_IR
1	1	1	1	1	1
2	2	1	1	1	0
3	2	1	1	1	1
4	1	1	1	1	0
5	2	1	1	1	2
6	2	0	1	1	1
7	2	0	1	3	1
8	2	0	1	4	1
9	2	0	1	2	0
10	2	0	1	2	0

Obs.	ALCHL_I	PROFIL_I_R	SUR_COND	VEH_INVL	MAX_SEV_IR
1	1	1	1	1	1
2	2	1	1	1	0
3	2	1	1	1	1
4	1	1	1	1	0
5	2	1	1	1	2
6	2	0	1	1	1
7	2	0	1	3	1
8	2	0	1	4	1
9	2	0	1	2	0
10	2	0	1	2	0